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ABSTRACT 

Project  Headstart  is  a  study  of  the  feasibility  of  military 
computer  maintenance  based  on  the  maintenance  of  a  pilot  site, 
the  Washington  Air  Defense  Sector  of  SAGE.  A  comparison  was 
made  of  the  IBM  experience  and  of  the  Air  Defense  Command 
experience  in  maintaining  this  site.  The  comparison  indicated 
that  military  maintenance  of  SAGE  computers  was  as  effective 
as  contractor  maintenance. 
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THE  ’’CONTRACTOR  PROBLEM” 

After  the  Korean  War,  the  Air  Force  found  itself  with  an  increasingly 
severe  manpower  problem:  how  to  get  enough  technicians  to  maintain  its 
growing  inventory  of  electronic  equipment.  Because  of  manpower  restrictions, 
the  Air  Force  was  not  able  to  fill  this  increasing  requirement  for  skilled  man¬ 
power  from  its  own  ranks.  Therefore,  it  was  forced  to  use  civilian  personnel 
for  much  of  its  electronic  maintenance  work.  The  DEW  Line,  for  example,  is 
almost  completely  staffed  by  contractor  personnel. 

Among  the  major  tasks  performed  by  contract  personnel  was  maintenance 
of  the  SAGE(AN/FSQ-7)  computers.  In  1956,  International  Business  Machines 
Corporation  (IBM)  was  awarded  the  job  of  maintaining  the  computers.  In  one 
respect,  SAGE  computer  maintenance  was  a  special  case;  because  digital  com¬ 
puters  were  new  to  the  Air  Force,  there  existed  no  Air  Force  career  field 
airmen  trained  to  maintain  them,  much  less  enough  men  qualified  to  maintain 
twenty-five  of  the  biggest  duplex  computers  ever  built. 

Contractor  maintenance  kept  the  machines  running,  but  it  had  drawbacks. 
High  cost  was  one,  though  it  probably  was  not  as  serious  as  some  have  suggested. 
Another  was  the  harmful  effect  on  the  military  operators'  morale  of  working 
with  men  being  paid  more  and  not  subject  to  military  restrictions.  However, 
the  biggest  problem  to  the  Air  Force  was  the  question  of  control:  what  would 
happen  if  war  broke  out  and  the  contract  technicians  went  to  their  families  rathe'r 
than  to  their  duty  station?  In  effect,  the  Air  Force  appeared  to  have  become 
dependent  on  men  over  whom  it  had  no  reliable  control  in  time  of  war.  ■  Con¬ 
sequently,  pressure  mounted  in  high  Air  Force  levels  to  arrest  or  reverse 
the  trend  to  maintenance  by  contract. 
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"BLUE  SUIT"  COMPUTER  MAINTENANCE 

In  late  1958,  the  Air  Defense  Command  (ADC)  drew  up  a  plan  for  military 
maintenance  of  the  SAGE  computers.  *  The  stated  goals  of  the  plan  were  to 
establish  military  control  over  the  maintenance  activities  and  to  reduce  the 
cost  of  SAGE  computer  maintenance.  On  the  vital  question  of  the  capability  of 
Air  Force  technicians  to  perform  computer  maintenance  effectively,  ADC  had 
received  assurance  from  IBM  that  the  airmen  could  do  the  job.  The  other  major 
concern  -  whether  enough  qualified  airmen  were  available  to  maintain  all  the 
SAGE  computers  -  was  a  matter  of  judgment  on  the  part  of  the  Air  Force.  Air 
Force  Headquarters  evidently  felt  that  there  were  enough,  and  in  January  1959 
authorized  ADC  to  carry  out  its  plan. 

The  overall  structure  of  the  ADC  plan  rested  firmly  on  the  pattern  set 
by  IBM  maintenance.  In  many  particulars,  the  statistical  card  room,  the 
training  program,  the  maintenance  handbooks  and  procedures,  and  the  supply 
program,  the  IBM  pattern  was  unchanged,  even  to  retaining  direct  IBM  manning. 
In  the  remainder  of  the  plan,  the  technician  job  classification  structure,  the 
shift  manning  and  rotation  practices,  and  the  manning  levels,  the  IBM  pattern 
was  used,  with  Air  Force  technicians  replacing  the  IBM  personnel. 

The  specific  manning  indicated  in  the  ADC  plan  reflected  both  the  IBM 
manning  structure  prevailing  when  the  plan  was  set  up,  and  a  body  of  Air  Force 
beliefs  on  maintenance  manning  requirements  that  appeared  reasonable.  The 
ADC  complement  was  set  at  58,  5  officers  and  53  airmen,  based  on  the  58  men 
IBM  used  in  1958  to  do  the  SAGE  computer  maintenance  job.  **  ^  In  addition 


*  Hq.  Air  Defense  Command  letter  to  HQ  USAF,  September  18,  1958;  Subject: 
SAGE  Computer  Maintenance  Proposal. 

**In  our  analysis  we  have  focused  solely  on  airmen  population.  None  of  our 
studies  is  concerned  with  the  officers. 
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to  the  military  maintenance  group,  the  ADC  plan  specified  13  IBM  technicians 

at  each  site  for  technical  support.  Although  the  reasoning  underlying  this 

support  was  never  formally  stated,  the  consensus  is  that  the  function  of  the 

back-up  is  to  provide  continuity  in  the  face  of  military  turnover,  to  smooth 

out  manpower  available  in  the  face  of  assumed  military  assignment  fluctuations, 

and  to  fill  in  for  the  assumed  difference  in  maintenance  quality  between  the 

58  IBM  men  and  the  58  ADC  men  assigned  to  each  site.  Further,  to  ensure 

that  ADC  would  have  enough  qualified  men  available  at  each  site,  the  plan 

specified  an  unusually  high  proportion  of  experienced  personnel;  the  proportion 

of  experienced  men  authorized  at  each  site  was  more  than  double  the  propor- 

[2] 

tion  prevailing  in  the  electronics  maintenance  career  field  as  a  whole. 

Finally,  the  plan  called  for  ADC  to  take  over  maintenance  of  the  SAGE  com¬ 
puters  at  the  rate  of,  roughly,  one  site  every  two  months.  The  problem  here 
was  that,  if  the  implementation  proceeded  on  schedule,  a  larger  number  of 
experienced  military  technicians  would  be  required  than  could  be  provided 
without  seriously  reducing  the  experience  level  throughout  the  rest  of  the  Air 
Defense  Command. 


WAADS  -  THE  PILOT  SITE 

Since  the  success  of  the  ADC  plan  rested  on  a  number  of  uncertainties, 
it  was  decided  to  try  the  plan  out  at  one  site  before  full  scale  implementation 
of  the  plan  had  proceeded  too  far  through  the  SAGE  system.  In  this  way,  plan 
modifications  could  be  made  on  the  basis  of  information  fed  back  from  the 
pilot  site.  In  addition,  should  the  pilot  site  experience  indicate  in  any  way 
that  the  ADC  plan  was  unfeasible,  a  return  to  contract  maintenance  could  be 
made  with  a  minimum  of  detriment  to  SAGE  operations. 
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The  Washington  Air  Defense  Sector  (WAADS)  was  chosen  as  the  pilot  site. 
The  MITRE  Corporation  was  selected  to  conduct  the  evaluation.  The  period  set 
for  the  evaluation  was  from  November  1959  to  August  1961;  from  the  date  that 
the  first  ADC  airmen  began  training  in  computer  maintenance  at  the  IBM  school 
in  Kingston,  New  York,  until  these  airmen  had  completed  one  full  year  of  com¬ 
puter  maintenance  at  WAADS. 

The  overall  evaluation  plan  and  the  specific  questions  to  be  asked  were 
obvious  from  the  nature  of  the  ADC  plan.  The  central  focus  had  to  be  on  an 
input -output  analysis  of  computer  maintenance  at  WAADS:  the  ADC  and  IBM 
technicians  assigned  to  WAADS  to  maintain  the  computer  would  be  compared; 
the  ADC  and  IBM  maintenance  performance  at  WAADS  would  be  compared.  Data 
from  the  remainder  of  the  sites  of  the  26th  Air  Division  of  SAGE  would  be  used 
to  provide  comparison  data  with  which  to  substantiate  the  conclusions  drawn 
from  the  WAADS  data.  Finally,  on  the  assumption  that  the  maintenance  effec¬ 
tiveness  at  any  site  was  a  function  of  the  technicians  at  that  site,  the  ADC 
technicians  would  be  checked  against  the  overall  ADC  technician  pool  from 
which  they  were  drawn  and  against  the  ADC  technicians  selected  for  the  sites 
to  be  manned  after  WAADS. 

In  this  way,  questions  then  being  asked  about  the  ADC  plan  could  be 
answered.  How  did  the  ADC  technicians  selected  for  WAADS  compare  with 
those  IBM  had  been  supplying?  If  there  were  differences  between  the  IBM 
and  the  ADC  technicians,  how  would  they  be  reflected  in  the  effectiveness  with 
which  the  computers  were  maintained?  How  would  the  Air  Force  technicians 
selected  to  maintain  the  other  SAGE  computers  compare  with  those  selected  to 
maintain  WAADS?  Finally,  what  measures  should  be  used  to  evaluate  the 
performance  of  technicians  maintaining  the  computers,  and  how  should  the 
effectiveness  of  computer  maintenance  at  a  SAGE  site  be  measured? 


THE  EVALUATION  PLAN 


To  compare  the  IBM  and  ADC  men,  standard  psychometric  data  were  used 
to  measure  technicians'  effect  on  SAGE  maintenance.  An  intelligence  test,  tests 
of  electronics  knowledge,  and  tests  of  reasoning  ability  had  been  given  to  all 
IBM  technicians  as  part  of  the  IBM  pretraining  selection  procedure.  These  test 
data  were  supplemented  by  the  training  grades  given  in  the  IBM  training  course. 
The  validity  of  these  test  data  and  training  scores  had  been  verified  by  IBM 
studies  comparing  them  with  on-site  supervisory  performance  evaluations. 

These  studies  showed  a  high  correlation  between  the  data  and  the  performance 
evaluations.  Arrangements  were  made  for  IBM  to  use  with  the  WAADS  airmen 
the  identical  tests  and  training  standards  that  had  been  used  with  IBM  technicians. 

Measuring  the  effectiveness  of  maintenance  rested  on  the  performance 

reliability  data  from  the  SAGE  computer  operations.  It  would  be  expected  that 

the  higher  the  computer  reliability,  the  higher  the  maintenance  effectiveness  of 

the  technicians.  However,  the  fact  that  the  SAGE  computers  at  each  site  were 

duplexed,  and  that  the  nonactive  (standby)  computer  at  a  site  might  be  operating 

in  a  number  of  different  modes  that  could  affect  the  down  time,  left  open  the 

possibility  that  this  generalization  might  not  be  true  for  SAGE  computer  main- 

[3] 

tenance.  Therefore  it  appeared  necessary  to  develop  a  method  of  analyzing 
the  site  computer  reliability  data  in  a  manner  that  would  make  it  possible  to 
isolate  for  each  time  period  the  particular  modes  in  which  the  standby  computer 
was  operating  and  the  time  that  it  was  in  each  mode.  It  was  also  necessary  to 
remove  from  the  comparison  those  data  that  reflected  situations  over  which  the 
maintenance  technicians  had  no  control  but  which  might  affect  the  reliability  of 
the  computer,  especially  periods  of  major  retrofit  activity  on  the  computer. 

(For  an  extended  discussion  of  SAGE  duplex  computer  reliability,  see  reference 
4.)  With  these  data  and  these  checks  on  the  data  determined,  a  comparison 
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could  be  made  between  IBM  maintenance  of  the  WAADS  computer  for  the  one- 
year  period  immediately  previous  to  ADC  assumption  of  maintenance  and  the 
computer  performance  at  WAADS  for  the  one-year  period  of  ADC  maintenance. 

It  was  then  necessary  to  determine  that  there  was  nothing  unusual  about 
the  particular  year  at  WAADS  during  which  the  ADC  technicians  were  main¬ 
taining  the  computer  and  that  there  was  nothing  unusual  about  the  manner  in 
which  the  reliability  data  had  been  gathered  and  reported  during  this  period. 

To  check  the  WAADS  data  on  the  first  point,  comparable  data  for  four  other 
sites  (Bangor,  New  York,  Syracuse  and  Sault  Ste.  Marie  Air  Defense  Sectors) 
were  analyzed  for  the  same  time  period  as  that  of  ADC  maintenance  at  WAADS. 
To  assure  that  the  data  collected  and  reported  by  WAADS  and  other  sites  for 
the  period  of  ADC  maintenance  were  accurate,  two  MITRE  representatives, 
one  of  whom  had  taken  the  IBM  computer  maintenance  course  with  the  WAADS 
group,  were  assigned  to  WAADS  from  April  1960  to  June  1961  and  worked  on 
the  analysis  of  site  reliability  data. 

Finally,  given  the  input -output  comparisons  from  WAADS,  one  question 
remained:  how  did  the  WAADS  technicians  compare  with  those  that  one  could 
expect  to  find  doing  SAGE  computer  maintenance  at  future  sites?  The  first 
requirements  here  were  measures  truly  reflecting  individual  maintenance 
performance.  Since  SAGE  computer  maintenance  was  a  group  activity,  no 
objective  individual  performance  measures  were  available.  The  only  recourse, 
and  one  valid  theoretically,  was  to  obtain  subjective  evaluations  of  the  perform¬ 
ance  of  each  technician  at  WAADS.  The  evaluation  technique  chosen  was  one 
in  which  each  technician  was  evaluated  by  each  of  his  peers  and  by  at  least  two 
supervisors.  This  technique  produced  composite  measures  of  the  performance 
effectiveness  of  each  technician.  These  composite  performance  measures  were 
correlated  with  previously  obtained  data  on  aptitude,  intelligence,  training 
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performance,  age,  time  in  service,  time  in  the  electronics  maintenance  field, 
and  grade  level  to  determine  which  factors  appeared  to  predict  individual  main¬ 
tenance  effectiveness  most  accurately.  With  these  predictors,  it  remained 
only  to  compare  the  WAADS  technicians  with  technicians  sent  by  ADC  to  main¬ 
tain  SAGE  computers  at  other  sites  and  with  the  pool  of  men  available  to  ADC 
for  maintaining  other  SAGE  computers  in  the  future.  This  comparison  would 
indicate  how  well  ADC  would  maintain  the  rest  of  the  SAGE  computers. 


ADC  COMPUTER  MAINTENANCE:  THE  FIRST  YEAR 
Unique  Conditions  at  WAADS 

The  first  year  of  ADC  computer  maintenance  was  performed  at  WAADS 
from  1  August  1960  through  31  July  1961.  Any  estimates  of  how  well  ADC 
technicians  can  be  expected  to  maintain  computers  in  the  future  use  the  ex¬ 
perience  of  this  period  as  the  basis  of  prediction;  but  the  manpower  provisions 
for  computer  maintenance  at  WAADS  were  in  certain  significant  respects  dif¬ 
ferent  from  what  can  be  expected  to  be  the  case  in  the  future. 

Some  of  the  differences  tended  to  give  WAADS  less  favorable  manning. 
WAADS  had,  by  definition,  a  group  less  experienced  in  computer  maintenance 
than  will  ever  again  prevail  at  a  SAGE  site  under  military  maintenance.  Spe¬ 
cifically,  two  basic  levels  of  technicians  make  up  the  maintenance  crews:  "units” 
technicians,  whose  training  and  experience  are  limited  to  either  the  central 
computer,  the  input-output  equipment,  or  the  display  equipment;  "systems" 
technicians,  drawn  from  the  better  "units"  technicians,  who  have  advanced 
training  and  experience  in  maintenance  of  the  overall  computer  system.  The 
WAADS  group  at  no  time  had  any  ADC  systems  technicians.  Further,  to  pro¬ 
vide  systems  technicians  for  future  site  manning,  ten  of  the  best  WAADS  units 
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technicians  were  sent  from  WAADS  to  IBM  for  system  training  in  February  1961 
and  were  not  returned  to  the  site  until  June  1961.  (See  Appendix  D) 

The  differences  that  may  have  tended  to  favor  the  WAADS  manning  are 
not  so  concrete.  Primary  among  these  is  the  fact  that  WAADS  was  assigned 
all  the  men  it  was  authorized.  This  situation,  100  percent  manning,  is  not 
characteristic  of  current  SAGE  manning  in  other  areas  of  electronic  mainten¬ 
ance  activity  (e.g.  radar  and  communications),  and  most  observers  acquainted 
with  the  ADC  personnel/ manpower  situation  believe  that  future  manning  levels 
in  computer  maintenance  will  probably  stabilize  in  the  area  of  60  to  80  percent 
manning  of  authorized  levels.  The  other  major  difference  that  may  have  favored 
the  WAADS  performance  was  that  all  the  technicians  and  supervisors  knew  that 
WAADS  was  in  the  spotlight,  and  therefore  reasonably  might  be  expected  to 
perform  better  than  if  they  were  not  working  under  experimental  conditions. 

Personnel  Input  to  WAADS 

What  kind  of  men  were  assigned  to  WAADS?  By  and  large,  ADC  sent  as 
good  a  group  to  WAADS  as  it  could  send,  though  these  men  were  not  equivalent 
to  the  IBM  men  that  they  replaced.  Comparing  the  WAADS  group  with  the  ADC 
career  field  group  from  which  they  were  selected  showed  that  the  WAADS  group 
had  more  maintenance  experience,  a  higher  average  electronics  aptitude,  and 
a  higher  average  grade  level.  ^ 

Since  aptitude  and  experience  are  traditionally  the  means  of  identifying 
the  best  technicians  in  the  Air  Force,  it  seems  clear  that  ADC  had  fielded  one 
of  its  best  teams  for  WAADS.  But,  as  ADC  had  expected,  these  men  did  not 
measure  up  to  the  IBM  men.  In  terms  of  electronics  knowledge,  reasoning 
•  ability,  general  intelligence,  and  training  performance,  the  WAADS  group 
measured  significantly  lower  (at  the  .  001  level)  than  the  IBM  men  then  assigned 
to  computer  maintenance  in  the  26th  Air  Division  of  SAGE  as  shown  in  Table  I. 


9 


#  Table  I 

Comparative  Test  Results  of  WAADS  and  IBM  Technicians 


WAADS 
(N  =54) 

IBM 
(N  =  2 1 4) 

X 

( 7 

X 

<7 

Electronics  Knowledge 

33 

.  6.6 

40 

3.  7 

Reasoning  Ability 

37 

6.9 

45 

3.4 

Intelligence 

110 

9. 1 

122 

6.5 

Examination  Average 

69 

12.  6 

79 

7.2 

I  2  5] 

All  differences  significant  at  the  .  001  level.  ’ 


Maintenance  Output  from  WAADS 

Since  the  ADC  maintenance  personnel  assigned  to  WAADS  (the  inputs) 
appeared  less  capable  on  the  average  than  the  IBM  men  that  they  were  replacing 
there  was  uncertainty  about  what  would  happen  to  the  reliability  of  the  SAGE 
computer.  Surprising  to  many,  SAGE  computer  reliability  at  WAADS  remained 
at  the  same  high  levels  established  when  IBM  performed  the  maintenance. 
Perhaps  of  even  greater  surprise  to  many  was  the  fact  that  the  ADC  maintenance 
performance  was  in  no  apparent  way  hurt  when  the  ADC  WAADS  group  lost 
nearly  20  percent  of  its  best  technicians  in  the  February-June  1961  period.  The 
reliability  data  are  given  in  Table  n. 

When  presented  with  an  apparent  anomaly  such  as  inputs  having  no 
correlation  with  outputs,  a  logical  first  step  is  to  check  the  data;  in  this  case 
the  reliability  data.  Had  the  data  been  accurately  recorded?  They  had.  Had 
the  ADC  WAADS  maintenance  been  performed  under  a  markedly  more  favorable 
pattern  of  operating  modes  of  the  standby  computer?  It  had  not.  Was  there 
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something  operating  from  August  1960  through  July  1961  that  made  the  WAADS 
computer  intrinsically  more  reliable  than  it  had  been  previously?  Nothing  in 
the  reliability  data  from  other  SAGE  computers  for  this  period  lent  support  to 
the  notion  of  increased  intrinsic  reliability,  nor  was  there  any  major  engi¬ 
neering  change  in  the  WAADS  computer  to  which  an  increase  in  intrinsic 
reliability  could  be  attributed. 


Table  U 

Comparative  Reliability  Data  for  WAADS* 


IBM 

1959 

1960 

ADC 

1960 

1961 

5  mo. 

7  mo. 

5  mo. 

7  mo. 

Percentage  of  Time 

Computer  was  Available 

99.89 

99.  88 

99. 81** 

99.96** 

Mean  Hours  to  Failure 

22.10 

31.61 

27.  65 

37.40 

Mean  Hours  to  Repair 

0.17 

1.47 

1.06 

1.03 

Unscheduled  Maintenance 
as  a  Percentage  of 

Total  Time 

3.03 

4.52 

3.65 

2.72 

♦See  Appendix  C. 

**  A  corresponding  shift  in  computer  availability  from  99.  81  to  99.  97  occurred 
at  SSMADS  (Sault  Ste.  Marie  Air  Defense  Sector)  during  the  same  period. 
This  similarity  is  significant,  since  SSMADS  became  operational  at  the  same 
time  that  the  ADC  group  assumed  maintenance  responsibility  at  WAADS.  It 
then  seems  that  this  shift  at  both  WAADS  and  SSMADS  can  be  attributed  to 
the  same  cause,  the  learning  that  occurs  as  a  new  maintenance  group  becomes 
familiar  with  its  job. 
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The  Meaning  of  the  Data 

If  the  data  and  conclusions  were  accurate,  perhaps  something  was  wrong 
in  the  premises  that  led  to  the  belief  that  maintenance  manning  inputs  ought 
always  to  be  correlated  with  machine  reliability  outputs.  In  the  case  of  WAADS 
computer  maintenance,  the  premises  that  were  faulty  were  so  fundamental  and 
so  implicit  in  the  ADC  plan  that  no  one  had  ever  really  questioned  them.  These 
premises  concerned  the  number  of  maintenance  men  needed  and  the  level  of 
their  capabilities.  The  principal  error  in  these  premises  resided  in  the 
difference  between  the  difficulty  of  the  maintenance  job  required  for  the  SAGE 
computers  in  1958  and  the  difficulty  of  the  maintenance  job  in  1961. 


In  August  1958,  when  the  ADC  plan  for  SAGE  computer  maintenance  was 
written,  IBM  had  far  less  maintenance  experience  with  the  SAGE  computer  than 
it  had,  for  example,  in  January  1961,  when  ADC  was  maintaining  the  computer 
according  to  the  manning  structure  of  the  ADC  plan.  This  difference  in 
experience  and  knowledge  was  reflected  in  the  number  of  men  that  IBM  required 
to  maintain  each  computer*,  as  shown  in  Fig.  1. 


Fig.  1  IBM  SAGE  Manning 
Experience 


Calendar  Year 


*  Letter  of  January  17,  1962,  from  D.  Simek  (IBM)  to  H.  W.  Adams 
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The  job  of  AN/FSQ-7  maintenance  had  become  simpler  than  the  ADC  plan 
had  assumed  it  to  be;  that  is,  when  the  ADC  plan  was  written,  SAGE  computer 
maintenance  was  a  58 -man  job;  when  the  plan  was  put  into  effect,  it  had  become 
a  42 -man  job,  and  is  now  a  32 -man  job.  The  change  in  job  difficulty  came 
about  because  the  IBM  experience  was  translated  into  simplified  maintenance 
and  diagnostic  procedures  that  were  incorporated  into  on-site  training  programs, 
improved  maintenance  manuals,  and  improved  diagnostic  maintenance  programs. 

Thus,  the  apparent  contradiction  of  the  better  IBM  inputs  having  no 
noticeable  correlations  with  WAADS  reliability  output  is  understandable.  The 
WAADS  experience  showed  that  53  ADC  men  with  IBM  back-up  and  very  little 
experience  could  keep  the  computer  running  during  the  August  —  December 
1960  period  about  as  well  as  48  IBM  men  had  done  during  the  previous  year. 

The  WAADS  experience  from  February  through  June  1961  showed  that  42  ADC 
men  with  IBM  back-up  and  some  experience  could  keep  the  computer  running 
as  well  as  IBM  had  ever  done.  Further,  on  the  point  of  manning  levels,  of  the 
IBM  men  assigned  to  back  up  the  ADC  men,  the  workload  data  indicated  that 
even  in  peak  periods  there  was  only  enough  work  for  about  ten  men  ,  J  though 
an  average  of  15  IBM  men  were  assigned  as  back-up  from  August  1960  through 
July  1961.  (See  Appendix  B) 

The  following  conclusions  were  made  about  site  manning  requirements: 

(a)  53  inexperienced  ADC  men  with  17  IBM  back-up  men  perform 
SAGE  computer  maintenance  satisfactorily. 

(b)  42  semiexperienced  ADC  men  with  15  IBM  men  and  no  system 
technicians  performed  the  computer  maintenance  job  as  well 
as  IBM  technicians  did  at  other  sites  (from  February  through 
June  1961 ). 


13 


(c)  In  case  (b)  above,  the  same  results  could  probably  have  been 
produced  with  5  fewer  IBM  men. 

(d)  If  the  ADC  men  included  men  with  system  training,  one  would 
expect  that  fewer  than  42  ADC  men,  given  10  IBM  back-up  men, 
could  do  a  satisfactory  job  of  SAGE  computer  maintenance. 

FUTURE  SITE  MANNING 

ADC  has  selected  technicians  for  three  sites  in  addition  to  WAADS:  the 
New  York,  Boston,  and  Syracuse  sectors  (NYADS,  BOADS,  and  SYADS).  If 
the  qualifications  of  these  groups  imply  as  good  maintenance  for  their  sectors 
as  the  WAADS  group  did  for  WAADS,  and  if  ADC  has  enough  qualified  men  to 
populate  the  rest  of  the  SAGE  sectors  in  similar  fashion,  then  we  feel  confident 
that  military  maintenance  of  SAGE  is  feasible. 

The  NYADS,  BOADS,  and  SYADS  groups  do  appear  to  have  the  qualifi¬ 
cations  necessary  for  maintaining  computer  performance  at  reliability  levels 

[  7] 

equal  to  those  that  ADC  achieved  at  WAADS.  In  a  previous  study  it  was 

shown  that  the  following  qualifications  of  the  WAADS  group  were  significantly 

correlated  with  maintenance  effectiveness:  the  technicians’  average  in  IBM 

training,  and  their  scores  on  the  electronics  knowledge  test  and  the  Otis 

intelligence  test,  both  administered  by  IBM.  Valid  comparative  data  for  WAADS 

and  the  other  sites  are  available  for  the  electronics  knowledge  and  intelligence 

[  8] 

measures  and  are  given  in  Table  HI.  These  data  show  no  significant 
differences  among  the  four  site  populations. 
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Table  HI 

Comparative  Test  Scores  of  ADC  Technicians  at  WAADS  and  Follow-On  Sites 


WAADS 

NYADS 

BOADS 

SYADS 

Number 

53 

55 

55 

56 

Electronic  Knowledge 

33.3 

33.9 

33.6 

33.8 

(Average) 

Intelligence 

110.2 

107.8 

108.1 

109.8 

(Average) 

The  examination  averages  for  WAADS  cannot  be  compared  with  those  for  the 
other  three  sites,  since  the  content  and  number  of  questions  in  the  WAADS 
examinations  were  different  from  those  in  the  NYADS,  BOADS,  and  SYADS 
examinations. 

Although  there  are  no  important  differences  in  terms  of  probable  mainte¬ 
nance  effectiveness,  the  WAADS  group  was  different  from  the  other  three. 
Briefly,  the  WAADS  group  was  made  up  of  far  scarcer  ADC  airmen  than  were 
the  NYADS,  BOADS,  and  SYADS  groups.  As  shown  by  Table  IV,  the  WAADS 

group  was  drawn  from  perhaps  the  scarcest  manpower  resource  in  the  Air 

[9] 

Force;  experienced,  high-ranking  electronics  technicians. 
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Table  IV 


Qualifications  of  ADC  Technicians  at  WAADS  and  Follow-On  Sites 


WAADS 

NYADS 

BOADS 

SYADS 

Percent  having  completed 
first  tour  of  duty 

96 

78 

56 

67 

Percent  Staff  Sergeant  or 
higher 

78 

55 

36 

45 

Percent  from  electronics 
maintenance  career  field 

94 

76 

72 

78 

Average  years  of  communi¬ 
cations -electronics  experience 

8.9 

5.  8 

3.5 

4.6 

Overall,  then,  the  Air  Defense  Command  has  selected,  for  the  follow-on 
sites,  men  who  are  not  in  such  critically  short  supply  as  those  who  went  to 
WAADS  but  who  should  do  as  good  a  job  of  computer  maintenance  as  the  WAADS 
group  did.  More  simply,  ADC  has  provided  as  much  potential  maintenance  but 
at  a  lower  cost. 
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APPENDIX  A 


ANNOTATED  BIBLIOGRAPHY  OF  MITRE  HEADSTART  PUBLICATIONS 


1.  NORAD  Description  and  Manpower  Requirements  for  1960-1965; 

Carla-Mae  Festa,  TM-2589,  December  30,  1959.  (Secret) 

A  description  of  the  NORAD  system  as  of  1960  and  1965,  with  an  over¬ 
view  of  the  system,  and  a  detailed  estimation  of  the  manpower  requirements 
of  the  system  for  these  time  intervals. 

2.  USAF  Contract  vs.  Military  Manning  Potential:  NORAD  Electronic  Tech¬ 
nicians,  1959-1965;  H.  W.  Adams,  TM-2663,  March  1,  1960. 

(Confidential) 

A  study  of  the  availability  of  experienced  technicians  proportional  to 
NORAD  manpower  requirements.  The  conclusion  is  reached  that  the  supply 
of  experienced  technicians  proportional  to  demand  cannot  be  expected  to 
increase  during  this  period. 

3.  ADC  Maintenance  of  SAGE  Computers  -  Current  Status;  H.  W.  Adams, 
W-257,  May  13,  1959. 

The  ADC  plan  for  assumption  of  SAGE  computer  maintenance  responsi¬ 
bility  is  discussed,  as  well  as  IBM  experience  in  maintaining  the  computers, 
the  IBM  personnel  and  organization  patterns,  and  the  assumptions  under¬ 
lying  the  ADC  plan. 

4.  Evaluation  of  ADC  SAGE  Computer  Maintenance  Plan:  Project  Headstart; 

H.  W.  Adams,  W-472,  August  31,  1959. 

The  decision  for  early  implementation  of  the  ADC  plan  at  the  Wash¬ 
ington  Air  Defense  Sector  (WAADS)  to  evaluate  the  ADC  computer  mainte¬ 
nance  plan  is  indicated.  Also  indicated  are  points  of  apprehension  about 
the  realism  of  the  plan.  In  addition,  an  overview  of  Project  Headstart 
objectives,  the  evaluation  of  the  ADC  plan  based  on  the  WAADS  experience, 
is  given. 
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5.  Observations  of  AN/FSQ-7  Maintenance  Activities;  C.  D.  Smith,  W-2564, 
October  14,  1959. 

Observations  of  actual  IBM  maintenance  activities  are  recorded  con¬ 
cerning  the  organization,  manning,  job  descriptions  and  methods  of  opera¬ 
tion  of  a  typical  IBM  SAGE  sector  maintenance  activity. 

6.  ADC  Electronic  Technicians:  Aptitude  Level  Distribution;  Linda  T.  Sprague, 
W-2555,  October  20,  1959. 

A  preliminary  report  on  a  study  of  the  quality  level  of  the  airmen  in 
ADC  who  formed  the  population  from  which  men  were  to  be  drawn  for  SAGE 
computer  maintenance.  The  conclusions  were  that,  over-all,  the  airmen 
had  a  high  aptitude  level  and  that  the  aptitude  level  of  the  more  experienced 
technicians  tended  to  be  higher  than  that  of  the  less  experienced  men. 

7.  Project  Headstart;  H.  W.  Adams,  TM-2516,  October  29,  1959. 

Agreements  reached  by  MITRE,  HQ  USAF,  HQ  ADC,  HQ  26th  Air 
Division  (SAGE),  HQ  WAADS,  HQ  AMC,  HQ  ATC  and  IBM  at  a  meeting 
held  October  13-15  concerning  the  evaluation  plan  for  Project  Headstart 
are  detailed. 

8.  Analysis  of  ADC  Electronic  Technician  Aptitude;  Linda  T.  Sprague, 

TM-2676,  March  25,  1960. 

A  final  report  on  the  aptitude  level  distribution  of  ADC  electronic 
technicians  previously  reported  in  W-2555.  The  conclusion  is  reached  that 
the  technicians  re- enlisting  tend  on  the  average  to  have  a  higher  electronics 
aptitude  than  those  leaving  the  service.  The  data  used  here  covered  a  com¬ 
plete  enumeration  of  the  pertinent  categories  of  the  30XXX  career  field 
in  1959. 

9.  Personnel  Inputs  of  IBM,  WAADS _and  ADC  Technicians  Compared; 

H.  W.  Adams,  A.  S.  Morton,  F.  P.  Gatling,  TM-2797,  June  17,  1960. 

The  aptitude  and  intelligence  scores  and  training  grades  of  IBM  com¬ 
puter  maintenance  technicians  of  the  26th  Air  Division  (SAGE),  of  ADC  air¬ 
men  assigned  to  WAADS,  and  of  the  ADC  pool  of  maintenance  technicians 
from  which  the  WAADS  groups  was  drawn  are  analyzed.  It  is  concluded  that 
the  WAADS  technicians  display  less  potential  competence  than  the  IBM  men 
they  are  replacing,  but  more  competence  than  could  be  expected  from  a 
random  selection  among  ADC  maintenance  technicians. 
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10.  Selecting  Computer  Maintenance  Technicians:  Preliminary  Report; 

A.  S.  Morton,  H.  W.  Adams,  F.  P.  Gatling,  TM-2819,  August  1,  1960. 

Using  aptitude,  intelligence  and  training  data,  a  method  of  selecting 
airmen  for  computer  maintenance  is  suggested  which  would  optimize  train¬ 
ing  performance  of  those  selected.  This  report  is  superseded  by  TM-3049. 

11.  What  Does  an  Airman  Cost?;  Carla-Mae  Festa,  H.  W.  Adams,  W-3138, 
August  12,  1960. 

Based  on  HQ  USAF,  RAND  and  Congressional  Hearings  data,  the  cost 
of  electronic  technicians  is  estimated  to  be  at  least  $7500  annually  per  man 
in  the  United  States  and  $7800  overseas. 

12.  Analysis  of  Field  Engineering  Training  Measures;  A.  S.  Morton,  TM-2847, 
September  13,  1960. 

The  grades  given  in  the  IBM  computer  maintenance  course  are  examined 
for  the  WAADS  group  and  for  a  representative  group  of  IBM  student  techni¬ 
cians.  It  is  concluded  that  the  examination  grades  are  the  most  meaningful 
grades,  and  that  the  WAADS  technicians  were  graded  according  to  the  same 
standards  as  the  IBM  students. 

13.  Selection  for  AN/FSQ-7  Maintenance:  301XX,  303XX,  304XX  AFS  Compared; 
H.  W.  Adams,  Carla-Mae  Festa,  TM-2891,  November  2,  1960. 

In  response  to  a  request  from  HQ  USAF,  the  training  performance  of 
the  WAADS  group  is  examined  to  see  whether  the  man’s  previous  position 
classification  was  significantly  related  to  training  performance.  It  was 
determined  that  the  only  significant  performance  differences  were  among 
airmen  selected  from  the  3015X  classification.  (This  difference  has  since 
been  shown  to  be  insignificant  in  terms  of  on-the-job  performance. ) 

14.  SAGE  Computer  Maintenance  Evaluation  Criteria;  H,  W.  Adams, 

A.  S.  Morton,  W-3482,  December  12,  1960. 

A  method  for  evaluating  the  effectiveness  of  maintenance  of  the  SAGE 
computers  is  developed.  Based  on  reliability  data  from  a  single  site  and 
control  data  from  similar  sites,  it  is  shown  that  maintenance  performance 
as  a  variable  can  be  isolated  from  other  factors  affecting  computer  relia¬ 
bility  data. 
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15.  The  AN/FSQ-7  Computer  Operator  Position;  W.  J.  Vibbard,  TM-2956, 
January  17,  1961. 

The  duties  and  function  of  the  "computer  operator"  position  at  a  SAGE 
sector  are  examined.  It  is  concluded  that  the  position  is  simply  that  of  the 
computer  console  coordinator  and  should  properly  be  assigned  to  the  com¬ 
puter  maintenance  activity  at  the  sector,  rather  than  to  the  maintenance 
analysis  activity  as  had  been  the  case.  It  was  further  recommended  that 
the  manning  for  this  position  be  drawn  solely  from  technicians  trained  in 
computer  maintenance. 

16.  Maintenance  Organization,  Manning  and  Functions;  L.  G.  Pacine, 

J.  H.  Proctor,  W-3806,  March  23,  1961. 

An  analysis  is  made  of  the  position  duties,  manning,  and  organization 
of  the  SAGE  computer  maintenance  activities.  The  recommendations  made 
are  that  the  WAADS  pattern  of  organization  be  followed  at  future  sectors, 
that  more  study  be  done  of  the  proper  mixture  of  airmen  technicians  and 
IBM  support,  and  that  certain  changes  in  the  position  of  the  IBM  site 
manager  be  made  at  future  sectors  when  ADC  assumes  maintenance 
responsibility. 

17.  Selecting  Military  Maintenance  Personnel  for  the  AN/FSQ-7  Computer; 

A.  S.  Morton,  I.  W.  Miller,  Jr. ,  TM-3049,  April  21,  1961. 

An  analysis  is  made  to  determine  the  maintenance  performance  ef¬ 
fectiveness  of  the  individual  technicians  of  the  WAADS  group.  Correlations 
are  done  to  determine  those  factors  in  the  individuals  background  which 
best  predict  maintenance  performance  effectiveness.  It  is  concluded  that 
the  only  significant  predictors  are  the  individual's  course  training  grades, 
his  intelligence  level,  and  his  electronics  knowledge  test  grade.  Experience, 
aptitude,  military  rank,  age  and  reasoning  ability  were  found  to  have  no 
significant  correlation. 

18.  Military  vs.  Contract  Maintenance  of  the  AN/FSQ-7;  H.  W.  Adams, 

A.  S.  Morton,  W-3887,  April  20  1961. 

A  summary  report  of  the  findings  of  the  first  six  months  of  military 
maintenance  of  the  WAADS  computer.  The  tentative  conclusions  are 
reached  that  ADC  maintenance  will  probably  be  satisfactory,  and  that  both 
ADC  and  IBM  manning  levels  could  be  reduced  without  detriment  to 
machine  reliability. 
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19.  Mean  Time  To  Failure  —  A  Measure  of  Intrinsic  Reliability  of  AN/FSQ-7 
Computers?;  W.  S.  Rouse,  W-3817,  March  28,  1961. 

Reliability  data  from  five  SAGE  sectors  are  analyzed.  The  conclusion 
is  reached  that  the  mean  time  to  failure  of  the  computers  is  not  a  measure 
of  their  intrinsic  reliability  alone,  but  is  significantly  affected  by  differences 
in  the  manner  in  which  the  machines  are  maintained  and  used. 

20.  AN/FSQ-7  Contractor  Technician  Assistance  Requirements;  L.  G.  Pacine, 

H.  W.  Adams,  J.  H.  Proctor,  TM-3105,  July  5,  1961. 

Data  on  IBM  assistance  to  ADC  technicians  at  WAADS  are  analyzed. 

It  is  concluded  that  the  work  to  be  done  could  just  as  well  be  done  by  10 
IBM  technicians,  rather  than  the  13  specified  in  the  ADC  plan  or  the  15-17 
assigned  to  the  WAADS  sector. 

21.  Airman  Assignment  Accuracy;  J.  H.  Proctor,  W.  S.  Rouse,  TM-3106, 

July  5,  1961. 

Data  from  the  ADC  26th  Air  Division  (SAGE)  are  analyzed  to  determine 
how  well  the  Air  Force  assigns  airmen  to  the  jobs  in  which  they  can,  by 
Air  Force  definition,  do  the  most  for  the  Air  Force.  It  is  concluded  that 
assignment  accuracy  runs  between  83  percent  for  the  more  highly  techni¬ 
cal  career  fields  and  91  percent  for  the  less  highly  technical  career  fields. 
The  computer  program  used  to  analyze  airmen  assignment  data  is  indicated. 

22.  ADC  Computer  Maintenance  Men:  WAADS,  NYADS,  BOADS,  SYADS; 

A.  S.  Morton,  W-4639,  January  25,  1962. 

Various  characteristics  of  the  computer  maintenance  technicians  as¬ 
signed  to  the  first  four  sites  maintained  by  ADC  airmen  are  analyzed.  It 
is  concluded  that  the  men  assigned  to  the  three  sites  after  WAADS  should 
perform  as  well  as  those  assigned  to  WAADS,  and  that  the  men  at  the 
follow-on  sites  do  not  draw  as  heavily  on  the  scarce  personnel  resources 
of  the  Air  Force  as  did  the  WAADS  group. 
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AN/FSQ-7  CONTRACT  TECHNICIANS 

The  ADC  plan  of  August  1958  for  assuming  the  maintenance  responsibility 
of  the  SAGE  AN/FSQ-7  computer  provided  for  5  officers,  53  airmen,  and  13 
IBM  contract  technical  services  (CTS)  personnel  at  each  site  to  handle  the 
maintenance  tasks. 

The  role  of  the  CTS  personnel  was  to  provide  technical  assistance  as 
required  and  to  provide  a  measure  of  continuity  of  knowledge  which  could  not 
be  guaranteed  by  the  Air  Force  because  of  its  rotation  policies. 

Data  on  the  amount  of  time  spent  by  CTS  personnel  in  maintenance 
assistance  at  WAADS  from  1  August  1960  through  31  May  1961  is  summarized 
in  the  Table  V. 

Fig.  2  illustrates  the  amount  of  assistance  provided  daily  by  the  CTS 
personnel  during  this  period. 


Table  V 


Summary  of  Reported  Assistance  Time  (Hours) 


Central 

Computer 

Display 

Input - 
Output 

Total 

Certified 
Man  Hours 

Percent  of 
Recorded  Time 
Spent  in 
Assistance 

Aug. 

120.2 

46.2 

54.4 

220.8 

2832 

7.8 

Sept. 

39.3 

52.5 

25.  8 

117.6 

2696 

4.4 

Oct. 

74.0 

105.5 

58.4 

237.9 

2656 

9.0 

Nov. 

390.5 

485.0 

70.7 

946.2 

2744 

34.5 

Dec. 

257.4 

160.5 

128.0 

545.9 

2784 

19.  6 

Jan. 

136.2 

423.6 

79.4 

639.2 

2912 

22.0 

Feb. 

291.4 

236. 1 

287.4 

814.9 

2672 

30.4 

Mar. 

394.5 

495.4 

183.5 

1073.4 

2992 

35.9 

Apr. 

265.1 

541.9 

150.2 

957.2 

2677 

35.8 

May 

j 

434.3 

575.5 

338.  8 

1348.  6 

2867 

47.0 

2402.9 

3122.2 

1376.  6 

6901.7 

27,832 

24.8 

Percent  of 
.  .  ,  35 

total 

45 

20 
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APPENDIX  C 


AN/FSQ-7  RELIABILITY  DATA 

The  data  used  in  MITRE’ s  evaluation  of  maintenance  effectiveness  were 
collected  directly  from  the  AN/FSQ-7  operations  logs.  The  operations  logs 
are  daily  reports  that  list  the  status  (active  or  standby)  and  condition 
(performing  air  defense,  not  performing  air  defense,  scheduled  maintenance, 
unscheduled  maintenance,  etc.)  of  each  of  the  computers,  and  the  length  of 
time  spent  in  a  particular  status  condition.  As  the  status  and/or  condition 
changes,  new  log  entries  record  the  change. 

The  basic  approach  used  in  the  data  collection  was  to  reduce  48  hours 
of  computer  operations  per  day  (24  hours  per  computer)  to  24  houllj  of  system 
operation.  The  system  operating  time  was  subdivided  according  to  mode  of 
operation.  Periods  of  extended  retrofit  were  excluded.  (Extended  retrofit  is 
defined  as  a  retrofit  whose  duration  is  such  that  it  requires  the  shut-down  of 
the  active  computer  for  preventive  maintenance  —  in  general,  a  retrofit  of  at 
least  a  40 -hour  duration. )  In  any  particular  mode  of  operation,  one  computer 
is  running  the  active  air  defense  program  while  the  other  computer,  the 
standby,  is  engaged  in  one  of  the  following  functions: 

Mode  A:  The  standby  computer  is  cycling  a  program  (DCS) 

which  permits  immediate  switchover  to  active  air  defense. 

Mode  B:  The  standby  computer  is  running  a  program  or  being 

♦ 

used  for  some  purpose  other  than  as  back-up  for  the  active 

computer. 

Mode  C:  The  standby  is  in  scheduled  maintenance. 


Mode  D:  The  standby  is  in  unscheduled  maintenance. 


Mode  E:  The  standby  is  in  retrofit. 

Computer  system  time  was  distributed  among  the  various  system  modes, 
generally  following  the  coding  used  in  the  operations  log. 

Having  determined  the  method  for  tabulating  reliability  data,  the  next 
point  was  to  determine  how  to  derive  a  measure  of  maintenance  effectiveness 
from  these  data.  It  was  first  necessary  to  isolate  those  effects  due  to  causes 
other  than  maintenance  from  those  properly  attributable  to  maintenance.  The 
maintenance  correlated  measures  were  those  associated  with  periods  of  Lost 
Air  Defense  (LAD)  on  the  active  computer,  or  a  period  of  unscheduled  mainte¬ 
nance  on  the  standby  computer.  In  both  cases,  the  cause  of  the  unforeseen 
computer  status  was  a  computer  failure:  a  failure  was  therefore  defined  as  a 
computer  malfunction  such  that  the  on-going  function  of  that  computer  must  be 
interrupted.  Since  the  function  of  #ie  maintenance  activity  is  to  prevent  such 
unforeseen  interruptions,  the  occurrence  of  a  failure  isolates  maintenance 
ineffectiveness,  thereby  measuring  in  inverse  fashion  the  effectiveness  of 
maintenance. 

Using  the  mode  distribution  of  system  time,  the  next  step  was  to  isolate 
the  amount  of  LAD  time  attributable  to  maintenance.  The  amount  of  time  that 
performed  was  checked.  If  the  total  did  not  equal  24 
hours,  the  logs  were  checked  for  the  reason.  LAD  due  to  nonmaintenance 
factors;  e.g.  ,  program  change,  simulated  noplex,  or  external  power  failure 
was  considered  good  time.  LAD  resulting  from  a  computer  failure  was  then 
checked  and  each  occurrence  was  designated  by  tallying  one  failure  and  re¬ 
cording  the  amount  of  LAD.  Further,  the  mode  of  operation  in  which  the  failure 
occurred  was  recorded  and  the  length  of  LAD  time  was  subtracted  from  the 
amount  of  time  that  the  standby  spent  in  that  mode. 


Next,  the  periods  of  unscheduled  maintenance  were  checked.  Each  period 
of  unscheduled  maintenance  that  was  followed  and  preceded  by  a  period  of 
something  other  than  unscheduled  maintenance  was  designated  as  representing 
one  failure,  unless  LAD  immediately  preceded  the  unscheduled  maintenance. 

In  that  case  the  failure  already  had  been  recorded.  It  should  be  noted  again 
that  the  parameters  used  in  our  evaluation  are  based  on  system  performance; 
therefore,  no  distinction  is  made  between  computers,  or  between  active  com- 
puter  and  standby  computer  failures. 

From  this  point  the  data  reduction  was  straightforward.  The  daily  work¬ 
sheets  were  summed  and  the  information  is  listed  in  Table  VI  (the  parameters 
MTF  and  MTR  are  explained  below). 

The  time  over  which  the  data  were  collected  was  then  divided  into  four 
periods,  with  the  major  division  corresponding  to  the  day  that  ADC  assumed 
maintenance  responsibilities  at  WAADS  (1  August  1960).  The  periods  are  as 
follows: 

Period  1  (IBM)  1  August  1959  —  2  January  1960 

Period  2  (IBM)  3  January  1960  —  31  July  1960 

Period  3  (ADC)  1  August  1960  —  31  December  1960 

Period  4  (ADC)  1  January  1961  —  31  July  1961 

WAADS  data  were  gathered  for  all  four  periods  to  give  data  for  one  full 
year  of  IBM  maintenance  and  for  one  full  year  of  ADC  maintenance.  BAADS , 
SYADS  and  NYADS  data  were  collected  for  the  same  one -year  period  of 
ADC-WAADS  maintenance  and  for  the  period  just  previous  to  this.  SSMADS 
became  operational  on  15  July  1960,  and  data  were  collected  for  the  period 
during  which  ADC  maintained  the  WAADS  computer.  Tables  VII  through  X 
were  then  prepared  from  the  data  in  Table  VI.  Table  XI  was  prepared  from  data 
in  Table  VI  plus  Mode  B  time. 
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Active  Availability 


Active  availability  is  the  percentage  of  time  that  either  computer  was 
available  for  performing  air  defense.  In  general,  the  higher  the  active 
availability,  the  better  the  maintenance. 


Active  Availability  = 


Total  Time  -  LAD 
Total  Time 


x  100 


Table  VII 

Active  Availability  Percentage* 


Time  Period 

WAADS 

BAADS 

SYADS 

NYADS 

SSMDS 

1 

89 

2 

88 

97 

91 

95 

3 

8! 

98 

90 

90 

81 

4 

96 

96 

92 

96 

97 

*99  omitted  in  each  case 
Standby  Unavailability 

Standby  unavailability  due  to  unscheduled  maintenance  is  the  ratio  of 
unscheduled  maintenance  time  to  total  time.  In  general,  the  lower  the  un¬ 
scheduled  maintenance  time,  the  better  the  maintenance. 


Standby  Unavailability  = 


Unscheduled  Maintenance 
Total  Time 


x  100 
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Table  VIII 


Standby  Unavailability  Percentage 


Time  Period 

WAADS 

BAADS 

SYADS 

NYADS 

SSMDS 

1 

3.  18 

2 

4.52 

2.00 

1.94 

4.08 

3 

3.65 

1.67 

1.  82 

1.38 

3.27 

4 

i 

2.72 

i.10 

2.43 

1. 15 

1.96 

Mean  Time  to  Failure 

Mean  time  to  failure  (MTF)  is  the  average  failure  rate  for  the  system. 
In  general,  the  higher  the  mean  time  to  failure,  the  better  the  maintenance. 

MXF  =  Total  Time  -  LAD 
Number  of  Failures 


Table  IX 

Mean  Time  to  Failure  (Hours) 


Time  Period 

WAADS 

BAADS 

SYADS 

NYADS 

SSMADS 

1 

21.  61 

2 

31.61 

60.27 

38.04 

27.81 

3 

27.  65 

95.98 

39.24 

40.76 

42.29 

4 

37.40 

65.21 

47.96 

65.21 

55.  89 

Mean  Time  to  Restore 


Mean  time  to  restore  (MTR)  following  a  failure  is  computed  in  Table  X. 
In  general,  the  lower  the  mean  time  to  restore,  the  better  the  maintenance. 


MTR 


Unscheduled  Maintenance  +  LAD 
Number  of  Failures 


Table  X 

Mean  Time  to  Restore  (Hours) 


Time  Period 

WAADS 

BAADS 

SYADS 

— 

NYADS 

SSMADS 

1 

.71 

2 

1.  47 

1.23 

.77 

1. 15 

3 

1.06 

1.62 

.75 

.61 

1.47 

4 

1.03 

.74 

_ ! 

1.20 

.78 

1.11 

Hours  of  Maintenance  per  100  Hours  of  Useful  Computer  Time 

This  parameter  is  a  measure  of  maintenance  input  to  computer  work 
output.  In  general,  the  lower  the  maintenance  time  input  ratio,  the  better 
the  maintenance.  It  is  calculated  as  follows: 

Maintenance  _  Scheduled  Maintenance  +  Unscheduled  Maintenance 

Effectiveness  Total  Time  -  LAD  +  Mode  A  Time  +  Mode  B  Time 
V - ^ - /  v - ^ - / 

Active  Running  Standby  Running 

Time  Time 


Table  XI 


Hours  of  Maintenance  per  100  Hours  of  Useful  Computer  Time 


Time  Period 

WAADS 

BAADS 

SYADS 

NYADS 

SSMADS 

1 

23.63 

2 

24.25 

19.63 

20.92 

22.04 

3 

23.17 

19.00 

21.46 

20.92 

21.38 

4 

23.04 

18.88 

22.34 

20.33 

i 

20.46 

100 
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APPENDIX  D 


AN/FSQ-7  systems  men 

Men  are  selected  from  among  the  "units”  technicians  to  receive  advanced 
training  in  computer  system  maintenance.  This  additional  training  covers  all 
areas  of  computer  maintenance,  and  on  completion  of  training  the  men  become 
’’systems”  technicians.  The  systems  technicians  coordinate  and  handle  common 
problems  that  arise  within  the  three  ’’units”  maintenance  areas;  i.  e. ,  input- 
output,  display,  and  central  computer. 

At  the  beginning  of  Project  Headstart,  the  only  technicians  at  WAADS 
were  those  trained  in  one  ’’unit.  ”  In  February  1961,  ten  Air  Force  WAADS 
technicians  were  selected  for  systems  training  on  the  basis  of  their  performance, 
training  and  on-the-job  results,  and  Air  Force  rank.  The  selection  chose  the 
men  equally  from  the  different  shifts  and  the  different  maintenance  areas,  as 
shown  in  the  Table  XU. 


Table  XII 

Technicians  Selected  for  Systems  Training 


Shift 

ABC 

DEF 

GHJ 

Quality  Control 

No.  Selected 

3 

3 

3 

1 

Maintenance  Areas 

I/O 

DD 

CC 

GMS 

QC 

No.  Selected 

2 

2 

3 

2 

1 

Air  Force  Rank 

A/2 

A/l 

s/s 

m 

Total  WAADS 

3 

9 

20 

Q 

n 

1 

2 

No.  Selected 

0 

1 

2 

1 

B 

i 

1 

2 

D-l 


Data  were  collected  on  the  training  grades  for  the  Air  Force  systems 
men  and  a  sample  group  of  IBM  systems  men.  However,  it  was  not  possible 
to  make  a  meaningful  comparison  between  the  two  groups  because  the  examin¬ 
ations  were  different.  Table  Xm  summarizes  the  statistics  for  the  systems 
course  for  each  group. 


Table  XHI 

Examination  Average  for  Systems  Course 


Exam  Mean 

Standard  Deviation 

Number 

IBM 

80.85 

6.  42 

58 

AF 

_ i 

82.53 

9.71 

10 

A  comparison  was  made  between  the  IBM  AN/FSQ-7  technicians  and  the 
Air  Force  WAADS  technicians  in  terms  of  ability  levels  and  previous  achieve¬ 
ment  in  the  units  training  course.  In  Table  XIV  three  groups  are  compared: 
a  sample  of  IBM  technicians,  the  WAADS  group  excluding  the  men  selected  for 
systems  training,  and  the  WAADS  men  selected  for  systems  training. 


Table  XTV 

Comparison  of  Ability  and  Performance  in  Units  Course 


Significance  of 
Difference 

AF  Systems  vs 

IBM 

WAADS 

A  F  Systems 

IBM 

WAADS 

Electronic  Knowledge 
(FEAT  I) 

40.2 

32.2 

37.9 

NS 

>.  02 

Reasoning  Ability 
(FEAT  U) 

44.4 

37.2 

38.9 

>.001 

NS 

Intelligence 

(OTIS) 

121.7  ! 

109.7 

112.7 

>.  001 

NS 

Exam  Grade 

79.3 

_ i 

66.5 

82. 1 

NS 

>.01 

Data  were  collected  on  the  training  grades  for  the  Air  Force  systems 
men  and  a  sample  group  of  IBM  systems  men.  However,  it  was  not  possible 
to  make  a  meaningful  comparison  between  the  two  groups  because  the  examin¬ 
ations  were  different.  Table  xm  summarizes  the  statistics  for  the  systems 
course  for  each  group. 


Table  XIU 

Examination  Average  for  Systems  Course 


Exam  Mean 

Standard  Deviation 

Number 

IBM 

80.85 

6.42 

58 

AF 

82.53 

9.71 

10 

A  comparison  was  made  between  the  IBM  AN/FSQ-7  technicians  and  the 
Air  Force  WAADS  technicians  in  terms  of  ability  levels  and  previous  achieve¬ 
ment  in  the  units  training  course.  In  Table  XIV  three  groups  are  compared: 
a  sample  of  IBM  technicians,  the  WAADS  group  excluding  the  men  selected  for 
systems  training,  and  the  WAADS  men  selected  for  systems  training. 


Table  XTV 

Comparison  of  Ability  and  Performance  in  Units  Course 


Significance  of 
Difference 

AF  Systems  vs 

IBM 

WAADS 

AF  Systems 

IBM 

WAADS 

Electronic  Knowledge 
(FEAT  I) 

40.2 

32.2 

37.9 

NS 

>.02 

Reasoning  Ability 
(FEAT  II) 

44.4 

37.2 

38.9 

>.001 

NS 

Intelligence 

(OTIS) 

121.7 

109.7 

112.7 

>.001 

NS 

Exam  Grade 

79.3 

66.5 

82.1 

NS 

>.01 

